Overview Scoring Rules

Bias

should be around 0.5 (between -1 and 1 according to Gunnar, but that seems just wrong)

Sharpness

smaller is better.

CRPS

generalisation of Brier score to continuous variables. Smaller is better.

LogS

Advantage: logarithmic Score penalises underestimating uncertainty heavily. I feel this is what we want.

Drawback: In contrast to the CRPS, the computation of LogS requires a predictive density. An estimatorcan be obtained with classical nonparametric kernel density estimation (KDE, e.g. Silverman1986). However, this estimator is valid only under stringent theoretical assumptions and canbe fragile in practice: If the outcome falls into the tails of the simulated forecast distribution,the estimated score may be highly sensitive to the choice of the bandwidth tuning parameter.In an MCMC context, a mixture-of-parameters estimator that utilizes a simulated sampleof parameter draws rather than draws from the posterior predictive distribution is a better

–> especially problematic I think if we work with traces and only small sample sizes?

Question do we now the posterior distribution of our draws?

DSS

Performance over time horizons

1 day ahead

Predictions

1 day ahead prediction

Scoring

Conclusion

Across the bench, Sparse AR seems the most reasonable (little bias, ok DSS and LogS)

AR1 seems to be very unconfident (and therefore performs well on LogS) AR1 seems to be downward biased.

1 day ahead metrics

horizon score model bottom lower median mean upper top sd
1 bias AR 1 0.0000000 0.1800000 0.4000000 0.4208571 0.6400000 0.9600000 0.2805722
1 bias Semi-local linear trend 0.0045000 0.2600000 0.5000000 0.4886122 0.7000000 1.0000000 0.2828632
1 bias Sparse AR 0.0000000 0.2600000 0.4400000 0.4429796 0.6000000 0.9400000 0.2539925
1 bias Student local linear trend 0.0000000 0.2800000 0.4600000 0.4940408 0.7400000 1.0000000 0.2887849
1 crps AR 1 0.0097440 0.0293893 0.0526451 0.0794650 0.0890898 0.3660674 0.0898345
1 crps Semi-local linear trend 0.0084990 0.0180820 0.0310165 0.0582366 0.0626547 0.2951930 0.0804311
1 crps Sparse AR 0.0101793 0.0239739 0.0382058 0.0670498 0.0746289 0.3242686 0.0869147
1 crps Student local linear trend 0.0078844 0.0175479 0.0276261 0.0574862 0.0627260 0.3423243 0.0799965
1 dss AR 1 -6.7070804 -4.2726994 -3.2469240 -2.7868620 -2.0327237 4.1355028 3.5341067
1 dss Semi-local linear trend -6.8608629 -5.4140323 -4.4642409 -3.1949170 -3.0414601 6.9085930 7.2741175
1 dss Sparse AR -6.2707872 -4.7646250 -4.0417495 -3.2725326 -2.6254400 3.2154416 4.1854044
1 dss Student local linear trend -6.9603421 -5.6213290 -4.5726281 -3.0854304 -3.0623496 8.5178083 8.6348196
1 logs AR 1 -2.3759070 -1.2771117 -0.8254659 -0.3512798 -0.2124687 3.7791279 3.3550183
1 logs Semi-local linear trend -2.4113148 -1.7454975 -1.2850647 0.1676840 -0.5942576 5.1365309 14.1584179
1 logs Sparse AR -2.2151640 -1.4355747 -1.0007693 -0.4215960 -0.4075420 2.3876052 4.8899705
1 logs Student local linear trend -2.5628558 -1.8588923 -1.3665089 0.6381316 -0.5829726 7.2153839 21.4649779
1 sharpness AR 1 0.0221096 0.0709010 0.1052553 0.1373040 0.1700109 0.4080446 0.1137602
1 sharpness Semi-local linear trend 0.0220793 0.0425032 0.0709581 0.0970184 0.1132235 0.3527295 0.0872100
1 sharpness Sparse AR 0.0273751 0.0597690 0.1020642 0.1307556 0.1481714 0.4327618 0.1075388
1 sharpness Student local linear trend 0.0201258 0.0389854 0.0597141 0.0908121 0.1074593 0.3644611 0.0861209

3 day ahead

Predictions

3 day ahead prediction

Scoring

Conclusion

AR1 seems very bad in terms of bias and everything.

Sparse AR is the best in terms of crps, AR1 the worst. Sparse AR also best with dss

–> take Sparse AR

All models have a tendency to be downwards biased, the local and semilocal ones tend to do a bit better.

3 day ahead metrics

horizon score model bottom lower median mean upper top sd
7 bias AR 1 0.0000000 0.0250000 0.2800000 0.3428108 0.5800000 1.0000000 0.3221013
7 bias Semi-local linear trend 0.0000000 0.1200000 0.4100000 0.4577838 0.8150000 1.0000000 0.3509869
7 bias Sparse AR 0.0000000 0.0650000 0.3600000 0.3907568 0.6400000 1.0000000 0.3230276
7 bias Student local linear trend 0.0000000 0.1400000 0.4400000 0.4797297 0.8200000 1.0000000 0.3453366
7 crps AR 1 0.0224146 0.0939478 0.1810374 0.3033660 0.3944736 1.1650918 0.3233669
7 crps Semi-local linear trend 0.0408044 0.1136825 0.2071928 0.3195261 0.4456299 1.1783405 0.3019611
7 crps Sparse AR 0.0411224 0.0869667 0.1639003 0.2659190 0.3420645 0.9273282 0.2865941
7 crps Student local linear trend 0.0366862 0.1256516 0.2213939 0.3495900 0.4473379 1.3712852 0.3557144
7 dss AR 1 -5.1511479 -1.9594018 -0.9830888 9.6924487 2.0741012 140.8036503 50.5344641
7 dss Semi-local linear trend -3.6826454 -1.9456928 -0.8493304 4.4328664 1.3250952 34.6275300 32.3297634
7 dss Sparse AR -3.7060228 -2.4520010 -1.2548468 2.4033523 1.0483922 34.4984516 12.6638976
7 dss Student local linear trend -3.6042707 -1.6218931 -0.7108610 6.2082494 1.2599417 48.4292621 48.2765050
7 logs AR 1 -1.6846635 -0.1356936 0.4478093 Inf 2.0985518 287.2871496 Inf
7 logs Semi-local linear trend -0.9164213 0.0317563 0.5867681 Inf 1.5800334 188.5458002 Inf
7 logs Sparse AR -0.8712581 -0.2524683 0.3474695 4.8517056 1.3516859 43.3777326 24.7326588
7 logs Student local linear trend -1.0542115 0.1465011 0.6724498 Inf 1.6191972 586.2568172 Inf
7 sharpness AR 1 0.0260282 0.1413464 0.2472176 0.3073751 0.3930559 0.9850034 0.2636063
7 sharpness Semi-local linear trend 0.0000000 0.1636543 0.2814537 0.3272691 0.4235071 0.8840668 0.2361977
7 sharpness Sparse AR 0.0601856 0.1622306 0.2385763 0.2984173 0.3420976 0.8644332 0.2239063
7 sharpness Student local linear trend 0.0000000 0.1693309 0.3359365 0.3761931 0.5000669 1.0860822 0.2989137

7 days ahead

Predictions

7 day ahead prediction

Scoring

Conclusion

Sparse AR still downwards biased local and semilocal doing better

crps: student local is best (it shouldn’t?), Semilocal and Sparse AR about equal. Sparse AR most consistent.

dss: student local doing best, then Sparse AR and semilocal

LogS: student local best, then Sparse AR wide ahead of the others.

Sparse AR most consistent.

3 day ahead metrics

horizon score model bottom lower median mean upper top sd
3 bias AR 1 0.0000000 0.0800000 0.3200000 0.3870222 0.6400000 1.0000000 0.3241965
3 bias Semi-local linear trend 0.0000000 0.1800000 0.4600000 0.4798667 0.7750000 1.0000000 0.3306509
3 bias Sparse AR 0.0000000 0.1600000 0.4000000 0.4207111 0.6400000 0.9800000 0.3044165
3 bias Student local linear trend 0.0000000 0.2200000 0.4500000 0.4877333 0.8000000 1.0000000 0.3268705
3 crps AR 1 0.0171687 0.0614331 0.1103108 0.1825217 0.2175170 0.8051064 0.2065734
3 crps Semi-local linear trend 0.0201682 0.0477773 0.0924898 0.1629336 0.1807126 0.8202050 0.2113492
3 crps Sparse AR 0.0206726 0.0506307 0.0928446 0.1560582 0.1823767 0.7172435 0.1885933
3 crps Student local linear trend 0.0163728 0.0477265 0.0863423 0.1690528 0.1992893 0.9294194 0.2305773
3 dss AR 1 -5.9276533 -2.9450615 -1.7709012 0.6555864 -0.2748510 25.7626906 12.1364841
3 dss Semi-local linear trend -5.1714026 -3.4990369 -2.2604842 1.0410709 -0.5360296 22.6703036 19.6737737
3 dss Sparse AR -5.1341712 -3.3923785 -2.2980771 -0.6108254 -0.7036617 16.5860866 7.2176870
3 dss Student local linear trend -5.4740366 -3.4486021 -2.4468675 7.1027040 -0.5750851 33.8384259 128.8999208
3 logs AR 1 -2.0104986 -0.6239897 -0.0319122 Inf 0.7560601 26.4391328 Inf
3 logs Semi-local linear trend -1.6811899 -0.7726919 -0.1932857 Inf 0.7073339 21.2044455 Inf
3 logs Sparse AR -1.7106809 -0.7202203 -0.2495037 1.3074993 0.5516765 16.3867174 9.0427728
3 logs Student local linear trend -1.8849301 -0.7960376 -0.2705402 Inf 0.7166684 71.4039034 Inf
3 sharpness AR 1 0.0259101 0.1093206 0.1716067 0.2187139 0.2829760 0.6810136 0.1803758
3 sharpness Semi-local linear trend 0.0358389 0.0953194 0.1487791 0.1922980 0.2379244 0.6989025 0.1579592
3 sharpness Sparse AR 0.0448981 0.1150896 0.1658032 0.2166656 0.2701764 0.6841201 0.1613564
3 sharpness Student local linear trend 0.0255382 0.0886775 0.1449848 0.1864862 0.2316276 0.6289547 0.1665941